A social force diagram of the My Car Won't Start Problem. This is a simple educational example of how if we analyze existing data that is often ignored (why solutions failed), then we can use that information to efficiently guide us to the root causes. This example was discussed in the first article in this series.
THE GOAL OF THIS ARTICLE is to teach you how to read and create social force diagrams.
A social force diagram is a high-level form of root cause analysis designed to analytically solve difficult large-scale social problems, such as democratic backsliding, climate change, systemic discrimination, and high inequality of income. This class of problems includes all the high-level hard problems of interest to progressives, so this tool should be of considerable interest to Kossacks working on hard problems.
This is the third article in a series dealing with the topic: Is it possible to take a more effective problem-solving approach to achieving progressive goals?
The first article introduced root cause analysis and then social force diagrams. The second article introduced causal diagrams, including feedback loops. This article covers social force diagrams in depth and builds on the first two articles.
A word of caution: This is not your average Kos article. It will take some concentration to follow. But we see no other way forward.
How science makes progress
THIS IS A STORY of scientific progress.
What do all scientific theories do? Only two things: Explain the past and predict the future.
In 1543 Copernicus did it with a comprehensive heliocentric theory of the solar system that explained how the Earth and planets revolve around the Sun, as opposed to the earlier geocentric theory where all heavenly bodies revolved around the Earth. The theory allowed astronomers to at last accurately predict planetary motion.
In 1687 Newton’s theory of universal gravity and the laws of motion (now known as classical mechanics) did it by explaining why all motion occurs. Now people could accurately explain past motion and predict future motion of any kind, such as why the tides occur, why Kepler’s laws of planetary motion work, and where a shot cannonball will go, given its starting speed, angle, size, weight, and air resistance.
Moving up to the twenty-first century, Nate Silver did it by inventing a new form of political election forecasting: weighted poll averaging. Before Nate pioneered the technique, poll based predictions were only slightly more reliable than horoscopes. In the 2000 US presidential election, the consensus, based on published forecasting models, was that Al Gore would win by as much as an 11 point landslide. Instead, Gore won the popular vote by 0.5% while George W. Bush narrowly won the electoral vote.
Fast forward to the 2008 election. Nate Silver (a former Kossak then writing as Poblano) became an overnight sensation by correctly calling 49 out of 50 states for the presidential election, and 35 out of 35 for senate elections. And then in 2012 he did it again, calling it correctly in all 50 states and 31 out of 33 senate seats. “Triumph of the Nerds: Nate Silver Wins in 50 States” proclaimed a Mashable headline. A Daily Beast article themed the Revenge of the Nerd described how:
The author of The New York Times–hosted blog FiveThirtyEight had correctly predicted the electoral lay of the land when all around him political pundits pronounced the race too close to call or maddeningly inconclusive.
Today in 2024, the nerds have won. Weighted forecasting models, based on
objective statistical analysis of poll data (with no thumbs on the scale via personal intuition), are the new norm.
How did the scientists and the nerds do it? By figuring out how to use existing data that was being ignored. Copernicus did it by using the hundreds of thousands of astronomical observations made by others to derive the parameters of the heliocentric model. Newton did it by using astronomical observations and prior theories to derive and prove the inverse square law of gravitation and the laws of motion. Nate Silver did it by using existing poll data in a new way. He statistically analyzed the past predictive accuracy of each polling firm, used that and date of polling to weight each firm’s poll results, and calculated a moving average for election result forecasts.
Let’s see if we can think like scientists and use existing data that is being ignored.
The great advantage of the superficial layer
Social force diagrams are organized into two layers: superficial and fundamental. The standard fill-in-the-blanks template looks like this:
The purpose of the superficial layer is to use existing data about solution failures to explain the past. Why did past solutions fail? Why did some do better than others? Starting at problem symptoms, you ask: WHY does this occur? to find the first intermediate cause. Then you ask the same question again: WHY does the first intermediate cause occur?
As you work down the causal chain from symptoms to first intermediate cause, second intermediate cause, and so on, you eventually arrive at the deepest intermediate cause. (The template has only one intermediate cause.) Then the question of WHY did that occur? allows you to at long last begin to penetrate into the fundamental layer, where the well-hidden root causes can be found.
At this point knowing where the deepest intermediate cause is gives you the world’s biggest clue about where to dig down the causal chain next. This is like the uncommonly good detective (or scientist) who uses ALL clues available, not just the ones everyone else is using. They know where to best dig next. No magic or super-human expertise is required.
Stop and think about this. Have you ever seen a story, analysis, or solution based on root cause analysis of the superficial layer? Probably not, because present approaches throw solution failure data away and ignore it. Yes, people try to learn from their solution failures and improve their solutions. But in the big seemingly impossible-to-solve problems they never put it all together into a comprehensive theory of WHY all past solutions are failing. The inevitable result is more trial and error and error and error. They don’t know where to dig since they don’t know where the deepest intermediate cause is.
Let’s further explore the advantages of the two layered design. The two layers are:
- The superficial (symptomatic) layer. Here intermediate causes are so easy to see they are erroneously assumed to be root causes.
- The deeper fundamental layer. Here, by understanding the problem’s feedback loop structure, its root causes may be found.
The superficial layer contains one or more intermediate causes. Some problems require multiple diagrams, since they contain multiple subproblems (defined by multiple symptoms) and thus multiple root causes. Difficult problems usually require construction of a feedback loop model to analyze the fundamental layer. Without root cause analysis of the fundamental layer, difficult problems tend to stay stuck in the superficial layer for a long time, as the Autocratic Ruler Problem (see below) did for thousands of years and many progressive problems are doing today.
This social force diagram was covered in the first article. Imagine how much sooner the problem could have been solved if a tool like social force diagrams had been used.
Social force diagrams are built by starting at problem symptoms and identifying the causal chain with “WHY does this occur?” questions until the root causes are found. As this is done, why past superficial solutions failed is diagrammed. This is critical knowledge, as it indicates the intermediate causes are indeed intermediate rather than root causes, and triggers insights about where to dig deeper. After the superficial layer of the problem is understood, the analyst digs down to follow the causal chain into the fundamental layer to find the problem’s true root causes.
Knowledge of the superficial layer and why past solutions failed is required for solving difficult problems. Karl Popper, who famously invented the principle that a theory or hypothesis is not scientific unless it can be falsified by measurement or experimentation, explains why (italics in the original):
We are always learning a whole host of things through falsification. We learn not only that a thing is wrong; we learn why it is wrong. Above all else, we gain a new and more sharply focused problem, and a new problem, as we already know, is the starting point for a new development in science.
(From All Life Is Problem Solving, Karl Popper, 1999, page 13.)
After the superficial layer is built, “a new and more sharply focused problem” that could not be seen before becomes not just visible, but crystal clear:
- What is the feedback loop structure that identifies the root cause of the lowest intermediate cause in the superficial layer?
- What is the high leverage point for resolving the root cause?
- What practical solutions can push on the high leverage point in a manner so well-engineered that the root cause stays resolved and the mode change is relatively permanent?
Because each question is so sharply focused, the answer landscape is relatively small and quickly searched.
The four forces of social force diagrams
Here’s a simplified social force diagram template for quick reference:
Social force diagrams focus on understanding four key forces: S, F, R, and new R. Superficial solutions (force S) fail because force S is always less than root cause forces (force R), indicated by S < R. By contrast, fundamental solutions (force F) can succeed because if the solutions are properly designed (especially their impact on feedback loop structure), force F can exceed force R, indicated by F > R. This leads to a systemic mode change, during which the old R is replaced by a new R. The new force R must be engineered to be strong enough and self-managing enough to permanently hold the system in the solved mode, due to the way force F fundamentally changes critical feedback loop structure and loop dominance.
If analysis shows no F > R exists (no resolvable root cause is found), the problem is unsolvable as defined. In this case problem definition (old symptoms) can often be relaxed to make the problem solvable, such as raising the maximum allowable global temperature rise for the climate change problem to make that problem solvable. Otherwise, the problem should be declared unsolvable and solution should not be attempted.
Once all four forces are understood and all key assumptions have been measured or tested with experimentation, the analyst has a sufficiently complete theory of the problem. Each of the four forces provides an explanatory tenet of the theory. This gives the social force diagram theory of problem behavior:
- Social Force S. Why past solutions failed (S < R).
- Social Force R. Why the problem occurs (force R is unresolved).
- Social Force F. Why fundamental solutions can be expected to succeed (F > R).
- New Social Force R. Why the mode change will be relatively permanent.
This suggests that any comprehensive theory of how to solve a difficult social problem must adequately explain all four forces. The above list thus serves as the four requirements for a comprehensive theory of a difficult large-scale social problem. The theory must identify the four forces and explain their causal structure. While this may seem like an excessively high bar, we see no other minimum set of requirements capable of specifying the information needed to solve problems of this class, due to their extreme complexity.
Application example: The Recurring Wars in Europe Problem
Now that we’ve explained the heavy duty technical stuff about how social force diagrams work, we can review an example of how to apply the tool. This is the Recurring Wars in Europe Problem.
Problem symptoms were recurring large-scale wars in Europe. First came World War One, which introduced the horrors of trench warfare, poison gas, and battlefield tanks. Losses were unprecedented. Total estimated deaths were 9 to 11 million military and 6 to 13 civilian, causing H.G. Wells to coin the expression that World War One was “The War to End All Wars.” That didn’t happen. Germany rebuilt its war machine, Hitler rose to power, and World War Two killed an estimated 80 million people. It was the deadliest war in human history.
At this point our social force diagram looks like this:
Let’s apply root cause analysis. WHY were wars recurring in Europe? What was the immediate precursor to war? The main cause was arms buildup with strong attack capability, as shown in the revised diagram.
What clues can we use to build the superficial layer of the diagram? How can we use data that has long been ignored to learn from the past?
Those who cannot remember the past are condemned to repeat it. — George Santayana, The Life of Reason, 1905.
First, is arms buildup with strong attack capability an intermediate cause or a root cause? We know it’s an intermediate cause because after the War to End All Wars, attempted solutions failed, leading to World War Two. They failed because they pushed on a low leverage point: direct ways to discourage attack. This was done with traditional superficial solutions like: peace treaties, defenses, royal inter-country marriage, etc. Repeated solution failure indicates superficial solutions. Adding these nodes to the diagram gives:
The superficial solutions failed because they pushed on a low leverage point where S < R. Whatever the deeper cause was, it was a stronger force than S.
Next, WHY did arms buildup with strong attack capability occur? The answer is as old as human history. We live in a world of limited resources, where the behavior of all species is driven by the evolutionary algorithm’s struggle for survival of the fittest. All living organisms must pursue a survival of the fittest strategy that maximizes their competitive advantage, because if they don’t they will lose out to those who do. This holds for individual organisms as well as groups, such as families, tribes, confederations, and nations. Members of a group cooperate to maximize their group’s competitive advantage over other groups, as well as to optimize their own benefits using those natural resources they own in the territory they control.
Thus the main cause of arms buildup with strong attack capability is pursuit of survival of the fittest strategies in order to maximize a state’s competitive advantage. That leads to this diagram:
Is this cause an intermediate cause or a root cause? Should it be in the superficial layer or the fundamental layer? Examination of the history of the problem shows it to be the main root cause, because it led to a fundamental solution that worked.
After the horrors of two successive world wars on European soil, problem solvers said never again, looked deeper, and intuitively (without use of formal root cause analysis) found the root cause, high leverage point, and fundamental solution. The high leverage point was tight inter-country coupling so if you harm another country you are harming yourself. The fundamental solution was the European Union, with the Benefits of Cooperation Feedback Loop. This caused a permanent mode change due to the feedback loop. Today no union member would even consider war against another member since that would be terribly self-destructive. The final diagram looks like this:
The fundamental solution worked because it pushed on a high leverage point where F > R. The new root cause forces resulted from careful design of the fundamental solution. Once member states took the first strong step toward tight inter-country coupling, a new reinforcing feedback loop began. The European Union started with market integration and proceeded to further integration and benefits (for most members) via a common currency, membership in NATO, military integration, open borders between member states, common policies on agriculture, fisheries, and regional development, etc.
Check out the desired mode on the right side of the diagram. This is the desired behavior. We want no more large-scale wars in Europe. That should happen, due to shared goals and a common good attitude. That in turn should happen due to the European Union’s tight inter-country coupling via free trade, common currency, free immigration, etc.
A comprehensive theory of problem behavior
The completed social force diagram describes (at the high level) a comprehensive theory of the Recurring Wars in Europe Problem. The theory does this by explaining the problem in terms of the four main causal forces involved:
- Force S. Why past solutions failed (force S < R). Due to lack of an appropriate analytical method, the fundamental layer of the problem was hidden by complexity. This caused problem solvers to be intuitively attracted to pushing on low leverage points with superficial solutions. This pattern of failure has occured on countless difficult social problems due to the Superficial Solutions Trap.
- Force R. Why the problem occurred (force R is unresolved). Arms buildup occurred and eventually led to war, due to the unresolved main root cause of pursuit of survival of the fittest strategies. Presence of force R is well supported in the biological and human behavior literature, where the principle of survival of the fittest plays a foundational role. Here the competing organism is a state. That force R was resolved indicates it was a root cause force.
-
Force F. Why the fundamental solution succeeded (force F > R). The fundamental solution of the European Union succeeded because it resolved the main root cause by adding a new feedback loop that was properly engineered.
-
Force New R. Why the mode change, and hence the solution, will be relatively permanent (force F causes force R to transition to New R). The fundamental solutions added the Benefits of Cooperation feedback loop. As the years went by the benefits of the European Union started piling up, just as they had for similar unions like the 6 states and 10 territories of Australia, and the United States, which grew from 13 to 50 states. More and more benefits has caused the European Union to grow from 6 to 27 nations. As the benefits grew, so did the strength of the union. Since the Benefits of Cooperation loop is a reinforcing loop, it will grow indefinitely until it reaches its limits. This mode will continue unless a new intervening force intrudes, such an ominous one we are seeing today: democratic backsliding.
Note how the tool of social force diagram gives an easy-to-understand, high-level theory that explains everything important. However, this was a retrospective problem so it was an easy problem to analyze. The fundamental solution was known.
The “That’s a wicked problem” argument
But in the top problems facing the world’s progressive problem solvers, the fundamental solutions are unknown. The difficulty of these long-unsolved problems (such as climate change, democratic backsliding, high inequality of income and wealth, systemic racial and gender discrimination, and war) is so high it’s off the chart. Many people consider these to be insanely difficult or impossible to solve wicked problems:
In 1973, design theorists Horst Rittel and Melvin Webber introduced the term "wicked problem" in order to draw attention to the complexities and challenges of addressing planning and social policy problems. Unlike the “tame” problems of mathematics and chess, the wicked problems of [governmental social] planning lack clarity in both their aims and solutions. In addition to these challenges of articulation and internal logic, they are subject to real-world constraints that prevent multiple and risk-free attempts at solving. As described by Rittel and Webber, wicked problems have 10 important characteristics:
1) They do not have a definitive formulation.
2) They do not have a “stopping rule.” In other words, these problems lack an inherent logic that signals when they are solved.
3) Their solutions are not true or false, only good or bad.
4) There is no way to test the solution to a wicked problem.
5) They cannot be studied through trial and error. Their solutions are irreversible so, as Rittel and Webber put it, “every trial counts.”
6) There is no end to the number of solutions or approaches to a wicked problem.
7) All wicked problems are essentially unique.
8) Wicked problems can always be described as the symptom of other problems.
9) The way a wicked problem is described determines its possible solutions.
10) Planners, that is those who present solutions to these problems, have no right to be wrong. Unlike mathematicians, “planners are liable for the consequences of the solutions they generate; the effects can matter a great deal to the people who are touched by those actions.”
I firmly disagree with the implication of the wicked problem concept, and by now I hope you do too.
Over and over, I’ve seen the equivalent of “That’s a wicked problem, so no wonder we can’t solve it.” argument trotted out. To those who deeply understand advanced forms of problem solving, that’s rubbish. Here’s why:
The real reason progressive activists, governments, and social scientists have been unable to solve members of the class of difficult large-scale social problems is NOT that they are inherently wickedly difficult or impossible to solve. It’s because problem solvers have not been using a form of root cause analysis designed for this class of problems. That is the gap this series of articles seeks to fill.
Difficult large-scale social problems are all solvable, with the right analytical tools and their correct application. The steady march of science will prevail over intuition and trial and error and error.
The next article in the series is here.
